Pre-segment point cloud data to run real-time shape extraction faster

ABSTRACT

A method, apparatus, system, and computer readable storage medium provide the ability to pre-segment point cloud data. Point cloud data is obtained and segmented. Based on the segment information, a determination is made regarding points needed for shape extraction. Needed points are fetched and used to extract shapes. The extracted shapes are used to cull points from the point cloud data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation under 35 U.S.C. § 120 of applicationSer. No. 14/536,266, filed on Nov. 7, 2014, with inventor(s) RonaldPoelman and Oytun Akman, entitled “Pre-Segment Point Cloud Data to RunReal-Time Shape Extraction Faster,” which application is incorporated byreference herein, and which application claims the benefit under 35U.S.C. Section 119(e) of the following and commonly-assigned U.S.provisional patent application(s), which is/are incorporated byreference herein: Provisional Application Ser. No. 61/901,069, filed onNov. 7, 2013, by Ronald Poelman and Oytun Akman, entitled “Pre-SegmentPoint Cloud Data to Run Real-Time Shape Extraction Faster.”

This application is related to the following and commonly-assignedpatent application, which application is incorporated by referenceherein:

U.S. patent application Ser. No. 14/536,232, entitled “OCCLUSION RENDERMECHANISM FOR POINT CLOUDS”, by Paulus Jacobus Holverda and RonaldPoelman, filed on Nov. 7, 2014, which application claims the benefitunder 35 U.S.C. Section 119(e) of Provisional Application Ser. No.61/901,067, filed on Nov. 7, 2013, by Paul Holverda and Ronald Poelman,entitled “Occlusion Render Mechanism for Point Clouds.”

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to point cloud data, and inparticular, to a method, apparatus, and article of manufacture forpre-segmenting point cloud data to run real-time shape extractionfaster.

2. Description of the Related Art

Point cloud shape/feature extraction is a highly desired technique tofacilitate the computer aided design (CAD) workflow. CAD designers don'twant to snap to point clouds but prefer to work with planes, corners,and center lines. Laser scanners sample a surface without any knowledgeof what is being sampling. Snapping to inferred geometry in real-time isproblematic—enough points need to be available to create a decentrepresentation, and fitting geometry is computationally intensive (on alarge and/or reduced set of points). Accordingly, what is needed is thecapability to quickly and accurately extract shapes/features from apoint cloud. To better understand these problems, a description of priorart point cloud systems may be useful.

Point clouds are often created by reality capture devices such as laserthree-dimensional (3D) scanners that measure a large number of points(e.g., from thousands to many billions of points [3D coordinates]) onthe surface of an object, and output a point cloud as a data file. Thepoint cloud represents the visible surface of the object that has beenscanned or digitized. With the increased usage of such reality capturedevices, large point cloud data sets are more frequently created forconsumption by design applications. The challenge that design softwarefaces is visualizing and using this data efficiently in theapplications. While the point cloud data set is often very large, thenumber of points an application is capable of handling for visualizationand other needs is a small fraction—a few million points, for example.Prior art methods fail to provide the ability to process the massivevolume of points, in real time (e.g., preserving a ten [10] frames persecond or faster rate) to read a small fraction of points that have theproperty of accurately representing the original data set without lossof detail or information that causes misrepresentation of the originaldata.

Point cloud visualization and applications are increasingly important indesign, especially due to the decrease in the price point of thetechnology. Point clouds can contain an enormous number of points. Oneof the major challenges is representing the set of points whileproviding the ability to extract a small subset that is highlyrepresentative of the spatial region of interest.

As described above, a point cloud is created using a laser beam/scannerthat scans objects/surfaces to obtain millions of points. For example,an image scanner on top of a car that drives through a city may obtainmillions upon millions of points. If such points are rendered in acertain environment, different viewpoints may be queried/requested anddisplayed to a user. However, with a point cloud, if a scene isrendered, the depth component may not map properly. As a result, avisualization of a point cloud may require one thousand (1000) times ofoverdraw before the correct visualization is achieved.

Prior art methodologies may attempt to select a subset of the points ina quick and efficient manner. One approach divides a volume of interestinto equal size rectangular 3D cells. Each of the thousands of cells maycontain millions of points. The issue arises as to how to determine howmany and which points to select from/in a cell. Further, point clouddata resulting from a scanner may have various artifacts that areundesirable. Accordingly, there is a desire to eliminate the scannerartifact, to normalize the point selection, and to obtain a uniformdistribution/real depiction of the distribution of the point cloud dataregardless of the location of the scanner. Many prior art approacheshave attempted to solve such problems. Such approaches range fromnearest neighbor based approaches to frequency domain based approaches.Concepts of downsampling, borrowed from image processing techniques havealso been incorporated in solutions to view large point data sets. Goalsof some techniques include noise removal and optimal representation, atthe expense of computation and with freedom to modify the original pointset.

In addition, as described above, to facilitate CAD workflows, a CADsystem may extract shapes/features from the points. It is desirable toperform such extraction in real time. If the extraction is attempted ona large set of points, it can be computationally intensive. However, ifthe extraction is attempted on a minimized set of points, the error rateincreases (e.g., shape extraction/fitting may not be accurate and/or maynot be possible due to an insufficient number/range of points).Accordingly, what is needed is a method and apparatus that reduces thenumber of points from a point cloud in order to properly, accurately,and efficiently extract shapes/features.

SUMMARY OF THE INVENTION

During the indexing process (e.g., the creation of an octree of scannedpoint cloud data), the point cloud data is segmented. Segmentinformation (e.g., the bounding box) of the various segments are storedinto the indexed file. When later reading the point cloud data (e.g.,during a snap to inferred geometry query), the segment information isused to only fetch the points necessary to perform a shape extraction.Once shape extraction is performed, the shapes can then be fetched inreal time in response to the desired user operation (e.g., a snapoperation).

BRIEF DESCRIPTION OF THE DRAWINGS

Referring now to the drawings in which like reference numbers representcorresponding parts throughout:

FIG. 1 is an exemplary hardware and software environment used toimplement one or more embodiments of the invention;

FIG. 2 schematically illustrates a typical distributed computer systemusing a network to connect client computers to server computers inaccordance with one or more embodiments of the invention;

FIG. 3 illustrates the logical flow for processing a point cloud inaccordance with one or more embodiments of the invention;

FIG. 4 illustrates an exemplary octree structure in accordance with oneor more embodiments of the invention;

FIG. 5 illustrates the pre-segmenting of point cloud data in accordancewith one or more embodiments of the invention;

FIG. 6 illustrates an exemplary segmentation based methodology that maybe used in accordance with one or more embodiments of the invention;

FIG. 7 illustrates a point cloud that has been segmented in accordancewith one or more embodiments of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following description, reference is made to the accompanyingdrawings which form a part hereof, and which is shown, by way ofillustration, several embodiments of the present invention. It isunderstood that other embodiments may be utilized and structural changesmay be made without departing from the scope of the present invention.

Overview

Embodiments of the invention pre-segment data from a point cloud andstore a bounding box of a segment into an indexed file that is usedduring shape/feature extraction.

Hardware Environment

FIG. 1 is an exemplary hardware and software environment 100 used toimplement one or more embodiments of the invention. The hardware andsoftware environment includes a computer 102 and may includeperipherals. Computer 102 may be a user/client computer, servercomputer, or may be a database computer. The computer 102 comprises ageneral purpose hardware processor 104A and/or a special purposehardware processor 104B (hereinafter alternatively collectively referredto as processor 104) and a memory 106, such as random access memory(RAM). The computer 102 may be coupled to, and/or integrated with, otherdevices, including input/output (I/O) devices such as a keyboard 114, acursor control device 116 (e.g., a mouse, a pointing device, pen andtablet, touch screen, multi-touch device, etc.) and a printer 128. Inone or more embodiments, computer 102 may be coupled to, or maycomprise, a portable or media viewing/listening device 132 (e.g., an MP3player, iPod™, Nook™, portable digital video player, cellular device,personal digital assistant, etc.). In yet another embodiment, thecomputer 102 may comprise a multi-touch device, mobile phone, gamingsystem, internet enabled television, television set top box, or otherinternet enabled device executing on various platforms and operatingsystems.

In one or more embodiments, computer 102 is communicatively coupled to,or may comprise, a laser scanner 134. Such a laser scanner 134 mayconsist of a field measurement device capable of producing a 3Drepresentation of present conditions through the collection ofindividually measured points. The set of all points collected andregistered with another after the scanning process is referred to as apoint cloud. Such a point cloud may be stored in data storage devices120/124, within the scanner 134, in memory 106, and/or in any otherdevice capable of storing such information. The laser scanner 134 mayutilize a variety of scanning methods including aerial, static, andmobile. Such laser scanning may scan millions of point in secondswithout climbing on equipment and/or conducting contact measurements.

In one embodiment, the computer 102 operates by the general purposeprocessor 104A performing instructions defined by the computer program110 under control of an operating system 108. The computer program 110and/or the operating system 108 may be stored in the memory 106 and mayinterface with the user and/or other devices to accept input andcommands and, based on such input and commands and the instructionsdefined by the computer program 110 and operating system 108, to provideoutput and results.

Output/results may be presented on the display 122 or provided toanother device for presentation or further processing or action. In oneembodiment, the display 122 comprises a liquid crystal display (LCD)having a plurality of separately addressable liquid crystals.Alternatively, the display 122 may comprise a light emitting diode (LED)display having clusters of red, green and blue diodes driven together toform full-color pixels. Each liquid crystal or pixel of the display 122changes to an opaque or translucent state to form a part of the image onthe display in response to the data or information generated by theprocessor 104 from the application of the instructions of the computerprogram 110 and/or operating system 108 to the input and commands. Theimage may be provided through a graphical user interface (GUI) module118. Although the GUI module 118 is depicted as a separate module, theinstructions performing the GUI functions can be resident or distributedin the operating system 108, the computer program 110, or implementedwith special purpose memory and processors.

In one or more embodiments, the display 122 is integrated with/into thecomputer 102 and comprises a multi-touch device having a touch sensingsurface (e.g., track pod or touch screen) with the ability to recognizethe presence of two or more points of contact with the surface. Examplesof multi-touch devices include mobile devices (e.g., iPhone™, Nexus S™,Droid™ devices, etc.), tablet computers (e.g., iPad™, HP Touchpad™),portable/handheld game/music/video player/console devices (e.g., iPodTouch™, MP3 players, Nintendo 3DS™, PlayStation Portable™, etc.), touchtables, and walls (e.g., where an image is projected through acrylicand/or glass, and the image is then backlit with LEDs).

Some or all of the operations performed by the computer 102 according tothe computer program 110 instructions may be implemented in a specialpurpose processor 104B. In this embodiment, some or all of the computerprogram 110 instructions may be implemented via firmware instructionsstored in a read only memory (ROM), a programmable read only memory(PROM) or flash memory within the special purpose processor 104B or inmemory 106. The special purpose processor 104B may also be hardwiredthrough circuit design to perform some or all of the operations toimplement the present invention. Further, the special purpose processor104B may be a hybrid processor, which includes dedicated circuitry forperforming a subset of functions, and other circuits for performing moregeneral functions such as responding to computer program 110instructions. In one embodiment, the special purpose processor 104B isan application specific integrated circuit (ASIC).

The computer 102 may also implement a compiler 112 that allows anapplication or computer program 110 written in a programming languagesuch as COBOL, Pascal, C++, FORTRAN, or other language to be translatedinto processor 104 readable code. Alternatively, the compiler 112 may bean interpreter that executes instructions/source code directly,translates source code into an intermediate representation that isexecuted, or that executes stored precompiled code. Such source code maybe written in a variety of programming languages such as Java™, Perl™,Basic™, etc. After completion, the application or computer program 110accesses and manipulates data accepted from I/O devices and stored inthe memory 106 of the computer 102 using the relationships and logicthat were generated using the compiler 112.

The computer 102 also optionally comprises an external communicationdevice such as a modem, satellite link, Ethernet card, or other devicefor accepting input from, and providing output to, other computers 102.

In one embodiment, instructions implementing the operating system 108,the computer program 110, and the compiler 112 are tangibly embodied ina non-transitory computer-readable medium, e.g., data storage device120, which could include one or more fixed or removable data storagedevices, such as a zip drive, floppy disc drive 124, hard drive, CD-ROMdrive, tape drive, etc. Further, the operating system 108 and thecomputer program 110 are comprised of computer program 110 instructionswhich, when accessed, read and executed by the computer 102, cause thecomputer 102 to perform the steps necessary to implement and/or use thepresent invention or to load the program of instructions into a memory106, thus creating a special purpose data structure causing the computer102 to operate as a specially programmed computer executing the methodsteps described herein. Computer program 110 and/or operatinginstructions may also be tangibly embodied in memory 106, data storagedevice 120/124 and/or data communications devices 130, thereby making acomputer program product or article of manufacture according to theinvention. As such, the terms “article of manufacture,” “program storagedevice,” and “computer program product,” as used herein, are intended toencompass a computer program accessible from any computer readabledevice or media.

Of course, those skilled in the art will recognize that any combinationof the above components, or any number of different components,peripherals, and other devices, may be used with the computer 102.

FIG. 2 schematically illustrates a typical distributed computer system200 using a network 204 to connect client computers 202 to servercomputers 206. A typical combination of resources may include a network204 comprising the Internet, LANs (local area networks), WANs (wide areanetworks), SNA (systems network architecture) networks, or the like,clients 202 that are personal computers or workstations (as set forth inFIG. 1), and servers 206 that are personal computers, workstations,minicomputers, or mainframes (as set forth in FIG. 1). However, it maybe noted that different networks such as a cellular network (e.g., GSM[global system for mobile communications] or otherwise), a satellitebased network, or any other type of network may be used to connectclients 202 and servers 206 in accordance with embodiments of theinvention.

A network 204 such as the Internet connects clients 202 to servercomputers 206. Network 204 may utilize ethernet, coaxial cable, wirelesscommunications, radio frequency (RF), etc. to connect and provide thecommunication between clients 202 and servers 206. Clients 202 mayexecute a client application or web browser and communicate with servercomputers 206 executing web servers 210. Such a web browser is typicallya program such as MICROSOFT INTERNET EXPLORER™, MOZILLA FIREFOX™,OPERA™, APPLE SAFARI™, GOOGLE CHROME™, etc. Further, the softwareexecuting on clients 202 may be downloaded from server computer 206 toclient computers 202 and installed as a plug-in or ACTIVEX™ control of aweb browser. Accordingly, clients 202 may utilize ACTIVEX™components/component object model (COM) or distributed COM (DCOM)components to provide a user interface on a display of client 202. Theweb server 210 is typically a program such as MICROSOFT'S INTERNETINFORMATION SERVER™.

Web server 210 may host an Active Server Page (ASP) or Internet ServerApplication Programming Interface (ISAPI) application 212, which may beexecuting scripts. The scripts invoke objects that execute businesslogic (referred to as business objects). The business objects thenmanipulate data in database 216 through a database management system(DBMS) 214. Alternatively, database 216 may be part of, or connecteddirectly to, client 202 instead of communicating/obtaining theinformation from database 216 across network 204. When a developerencapsulates the business functionality into objects, the system may bereferred to as a component object model (COM) system. Accordingly, thescripts executing on web server 210 (and/or application 212) invoke COMobjects that implement the business logic. Further, server 206 mayutilize MICROSOFT'S™ Transaction Server (MTS) to access required datastored in database 216 via an interface such as ADO (Active DataObjects), OLE DB (Object Linking and Embedding DataBase), or ODBC (OpenDataBase Connectivity).

Generally, these components 200-216 all comprise logic and/or data thatis embodied in/or retrievable from device, medium, signal, or carrier,e.g., a data storage device, a data communications device, a remotecomputer or device coupled to the computer via a network or via anotherdata communications device, etc. Moreover, this logic and/or data, whenread, executed, and/or interpreted, results in the steps necessary toimplement and/or use the present invention being performed.

Although the terms “user computer”, “client computer”, and/or “servercomputer” are referred to herein, it is understood that such computers202 and 206 may be interchangeable and may further include thin clientdevices with limited or full processing capabilities, portable devicessuch as cell phones, notebook computers, pocket computers, multi-touchdevices, and/or any other devices with suitable processing,communication, and input/output capability.

Of course, those skilled in the art will recognize that any combinationof the above components, or any number of different components,peripherals, and other devices, may be used with computers 202 and 206.

Software Embodiment

As described above, when attempting to render millions/billions ofpoints in real-time, a point cloud is processed. Such processing mayresult in a significant bottleneck. Consequently, as part of theprocessing, points that are not relevant to the screen spacerepresentation may be culled. To cull such points, systems may attemptto calculate planes/polygons with textures to represent the point cloudand cull the point cloud data using such planes/polygons (e.g., asdescribed in the copending application cross-referenced above). Oneissue that arises is how to quickly and efficiently recognize theplanes/polygons as a representation of the point cloud. Embodiments ofthe invention provide a methodology and system for facilitating theextraction of shapes/features/planes/polygons.

FIG. 3 illustrates the logical flow for processing a point cloud inaccordance with one or more embodiments of the invention. At step 302,point cloud data is obtained (e.g., from a laser scanner) and stored inan octree structure. The creation of an octree structure based onscanned point cloud data may also be referred to as the indexing of thepoint cloud data (as it is indexed into the octree structure). FIG. 4illustrates an exemplary octree structure in accordance with one or moreembodiments of the invention. An octree structure provides ahierarchical tree data structure for the point cloud data. In otherwords, an octree is a tree-based data structure for organizing andparsing 3D data. Such a structure enables spatial portioning,downsampling, and search operations on the point data set. Each octreenode has either eight children or no children. The root node 400describes a cubic bounding box 402 that encapsulates all points. Atevery tree level 404/406, the space becomes subdivided by a factor oftwo (2) which results in an increased voxel (i.e., a volumetric pixelthat represents a value on a regular grid in 3D space) resolution.

At step 304, at a relevant tree depth distance (determined by size oramount of points), a plane fitting methodology is run (e.g., based onRANSAC [RANdom Sample Consensus]). RANSAC is an iterative method used toestimate parameters of a mathematical model from a set of observed datathat contains outliers. Accordingly, at step 304, embodiments of theinvention iteratively attempt to find all of the planes and clip themwith the bounds of the relevant tree depth leaf node. Examples of such aplane fitting include least squares fitting (LSF) and principalcomponent analysis (PCA). In LSF, an iterative method is used to findthe best-fit plane with the least-squares constraint of the distancesfrom the scanned points to the plane (e.g., see Wang, M. and Tseng,Y.-H., 2004, Lidar data segmentation and classification based on octreestructure, International Archives of Photogrammetry and Remote Sensing,Vol. 20, B3, pp. 308-313 which is incorporated by reference herein).

More specifically, in step 304, an attempt is made to create and fitplanes on all the points within a predefined level of detail of theoctree. The number of planes is dependent on the amount of points thatconform to the plane fitting procedure and within a certain threshold.In this regard, the threshold may determine whether a maximum and/orminimum number of points are present in a predefined level. For example,a minimum number of points may be required in order to perform planefitting (e.g., one or two points may not comply with the minimumthreshold number of points while ten points may be sufficient).Similarly, too many points may be overly burdensome to perform the planefitting (e.g., consumes a disproportionate amount of processing and/ortime).

In view of the above, if level 404 is identified as the relevant level,the points within the nodes below level 404 would be used for the planefitting procedures. Thus, the appropriate octree level may be determinedbased on size (of the bounds of the level) or amount of points.

If based on the number/amount of points, the plane fitting process mayprogressively select sequential levels of the octree until asufficient/maximum number of points/per node for processing has beenreached. For example, suppose the plane fitting can process a maximumnumber of 25,000 points. If level 400 has one million points, the planefitting may proceed to level 404 which may have 125,000 points. Since125,000 is still beyond the maximum number of permissible points, theplane fitting may proceed to level 406 which could have 15,625 pointsper node. Since level 406 provides a number of points per node withinthe processing threshold, the plane fitting may identify level 406 asthe relevant level.

During the plane fitting, the detected planes are clipped with thebounds of the determined leaf node level of the octree to createrepresentative polygons.

Returning to FIG. 3, at step 306, additional processing may be performedon the planes/polygons, as necessary, in order to display/render thepoint cloud in an efficient manner.

While the steps performed in FIG. 3 may be useful and expedite theprocessing/culling points in a point cloud, performing such steps inreal-time on millions/billions of points may be problematic. Tofacilitate the processing, embodiments of the invention pre-segment thepoint cloud data during step 302 of FIG. 3.

FIG. 5 illustrates the pre-segmenting of point cloud data in accordancewith one or more embodiments of the invention. In this regard, the stepsof FIG. 5 may be performed during the indexing process (creation ofoctree of scanned point cloud data files).

At step 502, the laser scan data (e.g., point cloud data) is read.

At step 504, a segmentation based methodology is executed (e.g., a depthimage based segmentation methodology).

FIG. 6 illustrates an exemplary segmentation based methodology that maybe used in accordance with one or more embodiments of the invention.However, the invention is not limited to such a methodology and any typeof methodology may be utilized.

At step 602, the surface normal of each 3D point is calculated using itsspatial neighborhood.

At step 604, the surface normals are refined by assigning each point toa surface type such as smooth planar, jump edge, etc. and updating itsnormal information using its neighbors that belong to the same surfacetype. Refinement step 604 increases the normal precision especially onthe edge points.

At step 606, the 3D points are clustered into segments (e.g., using aregion growing methodology/algorithm). During region growing, two pointsare assigned to the same segment if they are spatially close(plane-to-point distances) and the angle between their surface normalsis smaller than some certain threshold. The clustering starts withselecting a seed point and recursively adding new neighbor(8-neighborhood) points to the seed point's segment if the neighborpoints satisfy the conditions explained above.

When the clustering/region growing step 606 is finished, all 3D pointsare assigned to a segment at step 608. FIG. 7 illustrates a point cloudthat has been segmented in accordance with one or more embodiments ofthe invention. Calculated segments may be shown in different colors(illustrated as different shades of gray in FIG. 7). Moreover, duringthe segmentation steps 606 and 608, various information/parameters suchas planarity, homogeneity, and bounding box are also extracted for eachsegment.

Returning to FIG. 5, from the results at step 504, the segmentidentification including the bounding box for the segment is stored atstep 506.

At step 508, an indexing structure (e.g., an octree structure) for thepoint cloud data is created/generated.

At step 510, the point cloud data and the segment information (e.g., thesegment identification and the bounding box) are stored in the indexingstructure. It may be noted that since the data is being read forprocessing into the octree, this additional pre-segmenting computationis cheap (i.e., the additional computation does not consume significantprocessing or time). The segment information (e.g., bounding box andsegment ID) may be stored in the octree/indexing structure or may bestored in a separate file. In addition, steps 506 and 510 may becombined into a single storage operation.

Once stored in the octree, the indexing process is complete and theindexing structure can be used to provide real-time shape extractionwhen working with the point cloud data (e.g., in steps 512-514) (i.e.,when extracting points and shapes from an indexing structure).

At step 512, when invoking a query (e.g., of the indexing structure) tosnap to inferred geometry, the segment information (e.g., the boundingboxes of the segments) is used to determine which points are needed forshape extraction.

At step 514, the determined points are fetched.

At step 516, the points are used to extract shapes (i.e., a shapeextraction methodology is executed using the fetched points).Embodiments of the invention are not limited to particular shapeextraction methodologies. Some methodologies may be non-real-time withno manual intervention, while others are real-time and less precise.Embodiments of the invention may combine the best of both online andoffline methodologies. For instance, least-squares based shapeextraction methods can be used to fit the best shape to the points fromthe same segments. While this approach is sensitive to outliers, it isfast and computationally cheap. On the other hand, computationally moreexpensive but robust and accurate methods utilizing non-linearoptimization methods or RANSAC+least-squares can be used when the shapeextraction algorithms are not required to run in real time.

At step 516, the extracted shapes are used to cull/retrieve points.

Thus, as described above, when working with point cloud data, users mayoften desire to perform a snap operation (e.g., move the cursor to aparticular point/location, attach objects to a point/object such as avertex, corner, plane, midpoint, center, etc.). Performing such a snapoperation in real-time is also highly desirable. However, working withmillions of points in point cloud data, such real-time operations areoften difficult if not impossible. Accordingly, geometry is ofteninferred from point cloud data. Snapping to inferred geometry isproblematic—enough points need to be available to create a decentrepresentation and fitting geometry is computationally intensive.Embodiments of the invention overcome such problems using thepre-segmenting steps described above. Once pre-segmented, the segmentinformation is used during the shape/feature extraction process. In thisregard, the shape/feature extraction process utilizes the segmentinformation to retrieve only those points that are needed to provide adecent and workable representation of the point cloud data. Morespecifically, once pre-segmented (e.g., steps 502-510), thecharacteristics of the segment and the segment parameters/informationcan be used to cull points and fetch shapes in real time (e.g., steps512-516). Such characteristics/parameters of the segment may include thesegment's size in 2D/3D (bounding box), homogeneity (if the segmentconsists of smooth surfaces), and planarity (if the surface is planar).

In view of the above, during the indexing process, the data ispre-segmented and the bounding box of the segment is stored into theindexed octree file. Thereafter, when utilizing the indexed point clouddata, the pre-segmented data is used to quickly determine and fetchpoints necessary for shape extraction.

CONCLUSION

This concludes the description of the preferred embodiment of theinvention. The following describes some alternative embodiments foraccomplishing the present invention. For example, any type of computer,such as a mainframe, minicomputer, or personal computer, or computerconfiguration, such as a timesharing mainframe, local area network, orstandalone personal computer, could be used with the present invention.

In summary, embodiments of the invention provide very accurate, highframe rate shape extraction on very large point clouds. Such shapeextraction allows CAD users to work with inferred geometry withoutfeeling the operation that provides the accurate real-time shapeextraction.

The foregoing description of the preferred embodiment of the inventionhas been presented for the purposes of illustration and description. Itis not intended to be exhaustive or to limit the invention to theprecise form disclosed. Many modifications and variations are possiblein light of the above teaching. It is intended that the scope of theinvention be limited not by this detailed description, but rather by theclaims appended hereto.

What is claimed is:
 1. A computer-implemented method for processingpoint cloud data, the method comprising the steps of: (a) pre-segmentingthe point cloud data by: (1) obtaining point cloud data, wherein thepoint cloud data comprises three-dimensional (3D) image data; (2)segmenting the point cloud data, into multiple segments, resulting insegment information, wherein the segment information comprises abounding box for each segment of the multiple segments; (b) invoking aquery to snap to inferred geometry of the point cloud data by: (1)determining, based on the bounding boxes in the segment information,points needed for shape extraction; (2) fetching the points needed forshape extraction; (3) extracting, in real time, one or more shapes basedon the fetched points; and (4) utilizing the extracted one or moreshapes to cull points from the point cloud data.
 2. Thecomputer-implemented method of claim 1, further comprising: creating andfitting planes on the point cloud data; culling the point cloud databased on the planes; and rendering the culled point cloud data; andwherein the pre-segmenting steps are performed during an indexing of thepoint cloud data.
 3. The computer-implemented method of claim 1, whereinthe segmenting comprises: calculating a surface normal of each point inthe point cloud data using a spatial neighborhood of each point;refining the surface normals by assigning each point to a surface typeand updating normal information using neighbors that belong to a samesurface type; and clustering the points in the point cloud data intosegments based on spatial proximity and surface normal angles.
 4. Thecomputer-implemented method of claim 1, wherein the segment informationfurther comprises: a segment identification for each segment of themultiple segments.
 5. The computer-implemented method of claim 4,wherein the segment information further comprises homogeneityinformation and planarity.
 6. The computer-implemented method of claim1, wherein the extracting comprises a least squares based shapeextraction.
 7. The computer-implemented method of claim 6, wherein theextracting further comprises a RANdom Sample Consensus (RANSAC) method.8. A non-transitory computer readable storage medium encoded withcomputer program instructions which when accessed by a computer causethe computer to load the program instructions to a memory thereincreating a special purpose data structure causing the computer tooperate as a specially programmed computer, executing a method ofpre-segmenting point cloud data, the method comprising the steps of: (a)pre-segmenting the point cloud data by: (1) obtaining, in the speciallyprogrammed computer, point cloud data, wherein the point cloud datacomprises three-dimensional (3D) image data; (2) segmenting, in thespecially programmed computer, the point cloud data, resulting inmultiple segments and segment information, wherein the segmentinformation comprises a bounding box for each segment of the multiplesegments; (b) involving a query to snap to inferred geometry of thepoint cloud data by: (1) determining, in the specially programmedcomputer, based on the bounding boxes in the segment information, pointsneeded for shape extraction; (2) fetching, in the specially programmedcomputer, the points needed for shape extraction; (3) extracting in realtime, in the specially programmed computer, one or more shapes based onthe fetched points; and (4) utilizing, in the specially programmedcomputer, the extracted one or more shapes to cull points from the pointcloud data.
 9. The non-transitory computer readable storage medium ofclaim 8, further comprising: creating and fitting, in the speciallyprogrammed computer, planes on the point cloud data; culling, in thespecially programmed computer, the point cloud data based on the planes;and rendering, in the specially programmed computer, the culled pointcloud data; and wherein the pre-segmenting steps are performed during anindexing of the point cloud data.
 10. The non-transitory computerreadable storage medium of claim 8, wherein the segmenting comprises:calculating, in the specially programmed computer, a surface normal ofeach point in the point cloud data using a spatial neighborhood of eachpoint; refining, in the specially programmed computer, the surfacenormals by assigning each point to a surface type and updating normalinformation using neighbors that belong to a same surface type; andclustering, in the specially programmed computer, the points in thepoint cloud data into segments based on spatial proximity and surfacenormal angles.
 11. The non-transitory computer readable storage mediumof claim 8, wherein the segment information comprises: a segmentidentification for each segment of the multiple segments.
 12. Thenon-transitory computer readable storage medium of claim 11, wherein thesegment information further comprises homogeneity information andplanarity.
 13. The non-transitory computer readable storage medium ofclaim 8, wherein the extracting comprises a least squares based shapeextraction.
 14. The non-transitory computer readable storage medium ofclaim 13, wherein the extracting further comprises a RANdom SampleConsensus (RANSAC) method.
 15. The computer-implemented method of claim1 wherein the pre-segmenting the point cloud is performed on the entirepoint cloud.
 16. The non-transitory computer-readable storage medium ofclaim 8 wherein the pre-segmenting the point cloud is performed on theentire point cloud.